For a particular judge, we can observe their release rate at each predicted risk decile. We can also observe the failure to appear rate. Question: Are these patterns of pretrial release decisions rationalisable as maximising expected utility at accurate predictions of failure to appear risk given some private information and a utility function that can vary across the four cells?
Do experts make systematic prediction mistakes based on the available information?
If so: - Which decision makers? - On which decisions? - In which ways are predictions systemically biased?
Three key identification challenges make this a very challenging econometric problem: 1. Decision-makers observe private information relevant to predicting the outcome 2. Unknown preferences that may vary across decisions 3. Missing data - outcomes only selectively observed for some of the decision maker’s choices
How can we tell if a decision maker’s choices reflect systematic prediction mistakes vs optimal behaviour at unknown preferences and information sets?
Strong assumptions or tailored structural models for each empirical setting: - Restrict preferences to be fixed across decisions and DMs - Observed choices as good as randomly assigned (Assume away private information) - Parametric models of private information
A unifying framework to analyse systematic prediction mistakes under weak assumptions on preferences and information sets in general observational settings.
We only observe the latent outcome if \(C=1\) \[Y := C \times Y^*\] * We observe \(P(Y^* \mid C=1, X )\) not \(P(Y^* | C=0 , X) \implies P(Y^* \mid X)\) partially identified. * Example: Only observe fail to appear if the judge grants bail. Failure to appear among all defendants only identified up to a set.
- No assumptions are placed on the distribution of \(V\) and we will model non-parametrically
\[
\underbrace{Q(Y^* \mid V, X)}_{\text{Posterior}} \propto \underbrace{Q(V \mid Y^*, X)}_{\text{Likelihood}} \underbrace{Q(Y^* \mid X)}_{\text{Subjective Beliefs}}
\] - Goal: Draw Inferences about DM’s subject beliefs \(Q(Y^* \mid X)\) using \((X, C, Y) \sim \mathcal{P}\), using data on their choices which reflect their posterior, leaving the private information and likelihood unspecified.
Suppose research partitions \(X = (X_{0}, X_{1})\)
\(u(c, y^*; x_{0})\): Payoff of choice \(c\) at outcome $y^* and characteristics \(x_{0}\)
Exclusion Restriction: Characteristics \(X_{0}\) directly effect both utility function and beliefs, whereas other characterises \(X_{1}\) and private information \(V\) only affect beliefs.
Suppose research partitions \(X = (X_{0}, X_{1})\)
\(u(c, y^*; x_{0})\): Payoff of choice \(c\) at outcome $y^* and characteristics \(x_{0}\)
Exclusion Restriction: Characteristics \(X_{0}\) directly effect both utility function and beliefs, whereas other characterises \(X_{1}\) and private information \(V\) only affect beliefs.
\(\implies\) Conditional on \(X_{0}\), variation in DM’s choice probabilities across \(X_{1}\) only reflect variation in posterior beliefs \(Q(Y^* \mid V, X)\) but not variation in utility function \(u(c, y^* ; x_{0})\)
Suppose research partitions \(X = (X_{0}, X_{1})\)
\(u(c, y^*; x_{0})\): Payoff of choice \(c\) at outcome $y^* and characteristics \(x_{0}\)
Exclusion Restriction: Characteristics \(X_{0}\) directly effect both utility function and beliefs, whereas other characterises \(X_{1}\) and private information \(V\) only affect beliefs. ### Example
Pretrial Release: Judges utility function only directly depends on
To show a DMs choices are consistent w/ expected utility max. at any accurate beliefs, linear utility, private information, it is sufficient to show that \(\forall x_{0}\) \[ \max_{\tilde{x}_{1}} \mathbb{E}\left[ \sum_{k} Y_{k}^* \mid C=1, X=(x_{0}, \tilde{x}_{1}) \right] \leq \min_{\tilde{x}_{1}} \mathbb{E}\left[ \sum_{k} Y_{k}^* \mid C=0, X=(x_{0}, \tilde{x}_{1}) \right] \] given \[ \underbrace{Y^* \mid \{ C=1, X \}}_{\text{point identified}} \text{ and } \underbrace{Y^* \mid \{ C=0, X \}}_{\text{not identified}} \] - We observe conditional distribution of \(Y^*\) for those released. We do not for those detained.
\(\implies\) DM’s choices are inconsistent w/ expected utility max. at accurate beliefs iff there exists no \(Y^* \mid \{ C=0 , X \}\) that satisfies the inequalities.
\[ \max_{\tilde{x}_{1}} \mathbb{E}\left[ \sum_{k} Y_{k}^* \mid C=1, X=(x_{0}, \tilde{x}_{1}) \right] \leq \min_{\tilde{x}_{1}} \mathbb{E}\left[ \sum_{k} Y_{k}^* \mid C=0, X=(x_{0}, \tilde{x}_{1}) \right] \] given \[ \underbrace{Y^* \mid \{ C=1, X \}}_{\text{point identified}} \text{ and } \underbrace{Y^* \mid \{ C=0, X \}}_{\text{not identified}} \]
With the assumptions so far, the DM’s choices are always consistent w/ EU max. at some linear utility function and accurate beliefs.
break
We therefore, need to make the following assumption
Econometric Assumptions to construct informative bounds on missing data
\[ \underline{\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \leq {\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \leq \overline{\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \leq \]
Behavioural Assumptions that exclude characteristics \(X_{1}\) and private information \(V\) from directly affecting the utility function.
With the assumptions so far, the DM’s choices are always consistent w/ EU max. at some linear utility function and accurate beliefs.
break
We therefore, need to make the following assumption:
\[ \underline{\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \leq {\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \leq \overline{\mathbb{E}} \left[ \sum_{k}Y_{k}^* \mid C=0, X \right] \]
We estimate these bounds through
Question: Are these patterns of pretrial release decisions rationalisable as maximising expected utility at accurate predictions of failure to appear risk given some private information and a utility function that can vary across the four cells?
And to do that, I use the quasi-random assignment of judges to cases and apply those instrumental variable bounds that I didn’t have the chance to talk through in more details to construct this curve in blue, which is an upper bound on the failure to appear rate among defendants detained by this particular judge at each defendant cell and each predicted risk decile. Now the identification results simply ask us whether there are any misrankings in this judge’s choices. And it turns out based on the point estimate, the answer is yes. This judge is releasing defendants at the top of the predicted risk distribution at a higher observed failure to appear risk while simultaneously detaining defendants at the bottom of the predicted risk distribution that have a strictly lower worst case failure to appear risk. They could exchange their choices and do strictly better no matter their utility function and private information.
These are point estimates; really, there should be standard errors around them. So formally, we would test
For each judge: 1. Calculate release rates by defendant cells 2. Observe failure rates for released defendants 3. Bound failure rates for detained defendants 4. Test for choice misrankings 5. Account for statistical uncertainty